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ABSTRACT 

An analysis was made of the methods being used to 
assess the Kentucky Education Reform Act (KERA) of 1990. Although no 
single assessment plan has been developed, various activities are 
aimed at studying both the implementation and the impact of the law. 
Four assessment efforts are described as conducted by the following 
groups: (1) the Office of Education Accountability, (2) bureaus of 
the Kentucky State Department of Education; (3) the school 
accountability system; and (A) the KERA evaluation governing board. 
All of these agencies have produced, or will be producing, important 
information about statewide educational reform. However, many 
questions about the KERA still remain to be investigated. The school 
accountability system is especially worthy of intensive study because 
of its unique features. The most recent accountability initiative is 
the KERA Evaluation Governing Board, 10 citizens charged with 
developing and implementing an evaluation of the KERA. Kentucky is 
making very extensive use of student performance assessment, rather 
than using standardized multiple-choice tests. Furthermore, results 
of performance assessment will weigh heavily on which schools get 
financial rewards and which schools receive sanctions. 
(Author/SLD) 
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Abstract 

An analysis was made of the methods being used to assess the Kentucky 
Education Reform Act (KERA) of 1990. Although no single assessment plan has 
been developed, a variety of activities are aimed at studying both the 
implementation and the impact of the law. Four assessment efforts were 
discussed at some length: a) the Office of Education Accountability, b) bureaus 
of the state department of education, c) the school accountability system, and 
d) the KERA evaluation governing board. All of these agencies have produced, 
or will be producing, important information about statewide educational reform. 
However, many questions about KERA remain to be investigated. The school 
accountability system is especially worthy of intensive study because of its 
unique features. Kentucky is making very extensive use of student performance 
assessment, rather than using standardized multiple-choice tests. Furthermore, 
results of performance assessment will weigh heavily on which schools get 
financial rewards and whl^.. schools receive sanctions. 
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The author was very tempted to limit this paper to the following single 
sentence: There is no plan for assessing the impact of the Kentucky Education 

Reform Act (KERA). The preceding was a very tempting statement because there 
is some element of truth to it. However, it also has some substantial 
inaccuracy. KERA does have a plan for its assessment, if the word "plan 11 is 
liberally interpreted and fully understood. Thus, I will not deliver a one- 
sentence paper and then go home! I aim in fhis paper to describe KERA and the 
current and future efforts to assess it. ... i he course of the discussion, I 
will explore the issues that have emerged in evaluating one state's educational 
reform effort, issues that are by no means unique, because they face educators 
in other states. 

I will place the experience of Kentucky in a national context by analyzing 
the state's attempts to assess educational reform. Many states have instituted 
educational reform efforts in recent years. In conjunction wita these efforts, 
different approaches have been developed to monitor the implementation and 
effectiveness of reforms. What approaches have been used or will be used by 
Kentucky? 

Origins of KERA 

In order to discuss the assessment of KERA, it is useful to briefly 
review the history and provisions of the law. The Kentucky Education Reform 
Act (KERA) of 1990 was the culmination of one of the most far-reaching state 
educational reform efforts in recent American history. The law was passed 
because the Kentucky Supreme Court declared in June 1989 that the entire state 
school system was unconstitutional. In judging a suit brought by a consortium 
of 66 low-income school districts, the court found that the state legislature 
had failed its responsibility to operate a statewide school system with 
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sufficient funding to assure equal educational opportunities to all children 
(Miller, Noland, & Schaaf, 1991). 

In arriving at its decision, the Kentucky Supreme Court used both social 
science data (e.g., drop-out rates, test scores) and financial data (Petrosko, 
1990). The social science data revealed that Kentucky compared poorly with the 
other 49 states and the states in the immediate region. The financial data 
revealed a system of taxation that was, on the average, low in level of support 
for schools, and quite variable from district-to-district. 

Section 183 of the Kentucky state constitution states: "The General 
Assembly shall, by appropriate legislation, provide for an efficient system of 
common schools throughout the state" (Rose vs. Council for Better Education, 
Inc., 1990, p. 205). Using a variety of data presented to it as evidence, the 
court found that Kentucky had an "inefficient" system of schooling-so much so 
as to be unconstitutional. 

The effect of the court decision was dramatic. In one stroke, all state 
statutes creating, implementing, and financing elementary and secondary 
education were invalidated. Thus, the legal basis for the state department of 
education, all local school boards, and laws on teacher certification, 
disappeared. 

The drastic nature of the Court's decision meant that the state 
legislature was forced to reform education. But determining the nature of the 
reform was very much open to interpretation. Since uneven funding was the 
basis of the lawsuit, state legislators could have restricted their activity to 
simply reforming school finance laws. They did not do this. Instead, a 
comprehensive law was passed with many educational changes, so many as to 
constitute a comprehensive overhaul of the entire state system of public 
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education. The necessity for reform of school funding became the opportunity 
to make a number of far-reaching reforms, in finance and in two other areas of 
education: governance and curriculum. 

In an earlier paper (Petrosko, 1990), I offered a few speculations about 
why reform took the course that it did in Kentucky. Briefly, I argued that a 
favorable conjunction of circumstances hel\ ad propel the reform effort. These 
included having: a) a governor with strong views about what kinds of changes 
should be made, b) a number of prominent citizens (from the media, business, 
academia) who had for years been lamenting the co .ition of education in the 
state and were eager to champion comprehensive reform efforts, and c) a state 
legislature with key members interested in systemic reform rather than 
piecemeal changes. In addition, reform occurred against the background of 
fears about the long-term economy in a state heavily dependent for revenues on 
two commodities with uncertain f utures--tobacco and high-sulphur coal. 
Undoubtedly, many other factors also contributed to the education reform 
effort; a truly comprehensive historical analysis of KEKA remains to be 
written. 

The changes mandated by the law, whether in school governance, in 
curriculum, or in finance, all had the same underlying purpose. The goal of all 
of these changes was to "assist students in acquiring basic skills, 
communication skills, understanding of governmental processes, knowledge to 
make economic, social and political choices, and other skills necessary to 
compete favorable with students in other states" (Miller, et al., 1991, p. 8). 
This statement of purpose and other statements of the rationale for KERA all 
point in the direction of the necessity to raise levels of student achievement. 
Achievement does not apply only to the basic skills; it also means problem- 
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solving and higher-order thinking. In the introduction to the law, the state 
legislature specified that schools must develop their students 1 abilities to 
"think and solve problems in school situations and in life; [and] connect and 
integrate experiences and new knowledge with what they have previously learned 
and build on past learning experiences to acquire new information through 
various sources" (Miller, et al., 1991, p. 28). This language came from a 
report issued by a study group of Kentucky citizens. The group had been 
charged by the governor to determine what Kentucky students will need to know 
to function successfully in the twenty-first century (Council on School 
Performance Standards, 1989). 

Provisions of KERA 

In order to effect improvement in student academic performance, an array of 
changes were prescribed. The scope and dimensions of these changes make the 
law difficult to succinctly summarize. KERA is not a unitary reform law with 
one or two prominent elements; it is, rather, a package of educational changes 
under a single title. 

To better grasp KERA as a whole, it is worth trying to look for some 
general characteristics of the law. Are there things that tie various 
provisions of the law together? I think several major themes are evident in 
KERA; I have identified four. 

One theme is the prime importance placed on high levels of student 
achievement and other student outcomes indicative of successful schooling. As 
was stated earlier, problem-solving and higher -order skills are given special 
prominence. The importance of higher-level types of achievement can be seen in 
several aspects of KERA: the stated curricular goals of the law, the 
curriculum framework developed for local districts, and the fact that the 



7 



5 

student assessment system includes testing procedures that aim at measuring 
more than basic skills. 

A second theme is that the individual school is viewed as the most 
significant social unit for educational change • Schoo 1 based dec is ion-making 
will be implemented in every district of the state. This means that a school 
council consisting of parents , teachers and the principal will have important 
powers regarding curriculum and school personnel. Futhermore, a new statewide 
student assessment system, using largely performance assessment, will have the 
school as the unit of analysis. Rather than emphasizing test scores of 
individual students, average scores in an entire school will be calculated and 
used in determining whether teachers and administrators in the school receive 
financial rewards or receive sanctions. 

A third theme is the attempt to increase professionalism and decrease 
crony ism . This is evident in several provisions. The Commissioner of 
Education, a gubernatorial appointee, is now the chief state school officer 
rather than the Superintendent of Public Instruction, an elective office that 
had been rife with political deal-making. A citizen's State Board for 
Elementary and Secondary Education governs education with the Commissioner. 
KERA has provisions against nepotism in hiring school employees and against 
other forms of financial favoritism. 

A final, and very discernable theme, is financial equity . This is not 
surprising, given that the law was passed because of a court suit brought by 
low income school districts. School finance changes are being put into place 
that will help equalize the expenditures per pupil of school districts 
throughout the state and to raise expenditures of ah districts. 

The provisions of the law were are organized into the categories curriculum, 
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governance, and finance. Some of the more significant of the law's requirements 
in each of these areas are summarized below. Following the summaries, Tables 
1, 2, and 3 provide more detailed lists of all of KERA's major provisions. 
Curriculum 

School-based decision-making was emphasized by the requirement that 
school councils be established in every school: councils involving parents, 
teachers, and the school principal. The councils have power to make policy for 
the school in such key areas as instructional materials, personnel, and 
curriculum. 

Ungraded primary school was established. Elementary schools will 
combine grades 1 through 3. Students will enter grade 4 only after successful 
completion of the primary school program. 

Student testing will evolve to a system dominated by performance 
assessment. There will be a reduction in the use of multiple-choice tests. 
Results of student performance assessment will contribute to a formula that 
determines educational rewards and sanctions for each school. 

Economic rewards (e.g., cash bonuses for teachers) will be given in those 
schools that increase their percentage of successful students. Sanctions will 
be leveled against schools with decreasing numbers of successful students. 
Governance 

A 12 member State Board for Secondary and Elementary Education was 

established. The 11 voting members are appointed by the governor, with the 
twelfth, non-voting member being the executive director of the Council on 
Higher Education. Board members are private citizens who, during their four- 
year terms, cannot be employed as professional educators. Appointment is 
without regard to political affiliation. The Board develops and adopts the 
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regulations that govern the state's 176 school districts and the state 
department of education* 

The office of Commissioner of Education, an appointed constitutional 
position, was established* The commissioner is the chief executive officer of 
both the State Board and the state department of education* The commissioner 
recommends and implements the policies of the State Board and directs the 
management of the state department of education* 

The Office of Education Accountability (OEA) was established under the 
Legislative Research Commission as an independent arm of the state legislature* 
The purpose of the Office is to: monitor implementation of KERA, review the 
state's system of school finance, verify the accuracy of school and state 
performance and investigate unresolved allegations of wrongdoing at the state, 
regional, or district level* 
F inance 

The Support Education Excellence in Kentucky (SEEK) fund was 
established, which guarantees the amounts of money spent per pupil throughout 
the state ($2,420.00 per pupil in 1991-92)* 

A minimal level of local financial support will be required (30 cents 
per $100 assessed valuation)* Furthermore, by July 1, 1994 all property will be 
assessed at 100% of fair cash value* 
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Table 1 



Major Curricular Provisions of the 1990 Kentucky Education Reform Act (KERA) 

Goals of education stresp higher-level thinking as well as basic skills, self- 
sufficiency, effective citizenship, and successful transition to adult life. 

A »odel curriculum framevork will be developed by the state and disseminated to 
local districts. 

Performance assessment will be implemented, including the use of writing exams 
and portfolios. Also, the following will happen. 

Monetary revards will be given to schools that increase their percentage of 
■'successful" students. Success will be determined by performance assessment 
results and non-cognitive factors (e.g. drop-out rates and attendance). 

Sanctions will be leveled against schools in which the percentage of 
successful students decreases . Measures could include removal of principals 
and teachers. 

Professional development opportunities for teachers will be increased. 

School-based decision making will be implemented. School councils consisting 
of parents, teachers, and the principal, will decide on curricular, personnel, 
instructional, and other matters. 

Pre-school programs for at-risk 4-year olds (and as many other 4-year olds as 
possible) will be initiated. 

Family Resource/Youth Service Centers will be established in schools within 
areas in which 20 percent or more of students qualify for free school meals. 

Technology in education will be emphasized to a greater extent. 

A primary school program will replace elementary school grades 1 -o 3. 

Kindergarten, minimum of half-day, five days a week, will be mandatory. 

Extended school services, by means of extended day s , weeks , or school years , 
must be provided to help all students meet learning outcomes. 

Age of compulsory attendance may be raised to 18 (upon recommendation of the 
Commissioner of Education). 




Table 2 

Major Governance Provisions of the 1990 Kentucky Education Reform Act (KERA) 

A 12-person State Board for Elementary and Secondary Education it established 
which will adopt policies for the department of education* 

A Commissioner of Education will be the chief executive officer of the State 
Board and the state department of education* 

The Start Department of Education will; 

assist districts with curriculum design, school administration, and finance, 

monitor the management of school districts, 

engage in research and planning, 

operate regional service centers with a primary focus on professional 
development of school district employees, 

establish an assessment center for school principals, 

establish a training program and assessment center for school district 
superintendents. 

An Education Professional Standard. Board is created to establish requirements 
for teacher certification, evaluate preparation programs, and issue teaching 
certi f icates . 

Members of local boards of education must: 

have at least a high school diploma or CED certificate, 
not have relatives employed by the school district, 

be limited to receiving campaign contributions of $100.00 from an individual 
and $200.00 from a Political Action Committee (PAC). 

Local school superintendents: 

ere responsible for ell personnel act ons in the district, 

can be terminated by a local board on y with approval of the Commissioner of 
of education, 

cannot hire relatives or board members' relatives to work for the local 
district • 

School employees must! 

not work for a school principal who is a relative, 

have a GED or high school diploma (classified employees), 

have the right of appealing a dismissal to a three-member tribunal 
appointed by the State Commissioner* 

I™ of f \ C h.°f ^ U f\ ti0n AccommUbiXlty (OEA) i. established as an independent 

1§ t0 "° nit0r W «d the 
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Table 3 

Major Finance Provisions of the 1990 Kentucky Education Reform Act (KERA) 

The Support Education Excellence in Kentucky (SEEK) state support fund is 
established to ensure, among other things: 

a guaranteed amount of money is spent per pupil, 

curriculum and governance provisions of KERA are funded (e*g., educating 
at-risk children, funding professional development for teachers). 

Minimal levels of local support are mandated: 

local tax rates will be 30 cents per $100 on property; 

utilities, including cable television, may be taxed. 

Ail real property is to be assessed at 100 percent of its cash value by 
July 1, 1994. 

No teacher will be paid below the minimum statevide salary schedule adopted in 
the state budget. 

For schools that do not have school-based decision making, saxiaus class sizes 

(in terras of numbers of pupils per class) are: 

Primary grades: 24 

Grade 4: 28 

Grades 5 and 6: 29 

Grades 7 through 12: 31 

For middle anJ secondary schools that do not have school-based decision making, 
the maximum teaching load will be 150 pupil hours per day. 

The State Board of Elementary and Secondary Education will establish a program 
to assist teachers in obtaining computers for their personal use. The program 
will include training provided by the vendor. 
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Kentucky Accountability Mechanisms for CERA 

At the outset of this paper I stated that one can plausibly argue there is 
no plan for assessing the impact of the Kentucky reform law. In the strict 
sense of the word "plan" this is true. There is no written document that 
presents a detailed set of goals, objectives and procedures to comprehensively 
assess KERA. However, it is also true that a variety of current and future 
activities are aimed at assessing both the implementation of the law and its 
impact. By drawing upon the sum total of these activities, policy makers, now, 
and in the future, can avail themselves of extensive data about the effects of 
the law. 

The source of these data are mechanisms (i.e., offices, positions, 
agencies, procedures) that are either: a) mandated by the law itself, 
b) natural outgrowths of past procedures, or c) results of future organizations 
that are now being formed. Four of these mechanisms are discussed below. 
Office of Educational Accountability (PEA) 

As mandated by the governance section of KERA, the Office of Educational 
Accountability (OEA) was established. OEA is under the Legislative Research 
Commission, a service agency of the Kentucky state legislature. OEA is defined 
as *'an independent arm of the legis lature 11 (Miller, et al., 1991, p. 16). The 
Office of Educational Accountability is charged with these tasks: monitoring 
the implementation of KERA. reviewing the state's system of school finance, 
verifying the accuracy of school district and state performance, investigating 
unresolved allegations of wrongdoing at the state, regional, or district level 
and reporting to the Legislative Research Commission (Miller, et al., 1991). 

OEA is considered the "watchdog 11 agency of school reform. The office places 
a relatively high priority on enforcing certain governance provisions of the 
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law. Specifically, KERA prohibits nepotism in hiring school employees and 
forms of inancial favoritism. 0E0 operates a toll-free telephone line that 
can be used to report allegations of such wrongdoing. In addition, the office 
employs several persons who investigate those allegations that are reported. 

The Education Accountability office engages in other forms of monitoring. 
For example, when personnel in the state department of education write 
regulations to implement a provision of KERA, OEA might be consulted to render 
a judgment on whether the regulations are consistent with the intent of the 
law. 

Department of Education Bureaus 

After KERA was passed, the Kentucky Department of Education was reorganized. 
New offices were established and new personnel were hired. Important changes 
were made, but continuity was maintained alsoj it is the nature of every state 
department of education to engage in systematic oversight of statewide 
programs. In what might be called a natural outgrowth of past procedures, the 
bureaus of the current state department of education perform activities that 
involve monitoring the school reform law. This is not really different than 
what the pre-1990 department of education did, or what any state department of 
education does. Only the content of the monitoring differs. 

The current Kentucky Department of Education has three bureaus: Learning 
Support Services, Learning Results Services, and Management Support Services. 
In addition, several offices and divisions (e.g., fiscal and strategic 
planning, legal services) report to deputy commissioners. All of these bureaus 
and offices strive to meet yearly objectives, and many of the latter are 
directly relevant to the education reform law. 

Below is an example of a yearly objective and a report on its status. The 
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example is from an August 1992 report of the Learning Support Bureau of the 
Department of Education. 



Annual Obiective 



Degree of Accomplishment 



A. By June 30, 1992, 350 schools 
will have successfully imple- 
mented school-based decision 
making indicated by student 
performance and participant 
surveys obtained during the 
1991-92 school v ear. 



Partially completed: As of June 8, 1992 
approximately 481 schools in Kentucky 
have implemented School-Based Decision 
Making. Surveys are being completed 
at this time to gauge success levels 
o f counc i Is • 



(Kentucky Department of Education, 1992, p. 10) 



The kind of monitoring and assessment performed by the department often 
involves obtaining data to determine whether schools, school districts, and "he 
department itself has complied with directives related to the law. The example 
cited above is a case in point. Other examples could be cited. For example, 
the Department requires local districts to file plans to carry out certain 
state-mandated objectives. For the most part, department monitoring might be 
called "bureaucratic oversight. 11 I am using bureaucratic in a literal rather 
than pejorative sense. 
School Accountability System 

In terras of an organizational chart, this accountability mechanism could 
have been classified with the category just discussed, the state department of 
-education. School accountability falls under the domain of the Learning 
Results Services Bureau of the department, and has its own associate 
commissioner. I made this a separate category because school accountability 
has a large profile, a large budget, and has many important implications for 
assessing KERA as a whole. The degree of importance placed on this part of the 
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reform law is revealed in the following quote, which comes from a citizen's 
guidebook to KERA published by the Legislative Research Commission, an agency 
of the state legislature* 

How will we know if education reform is working? 

Beginning in the 1991-92 school year, students will be tested in 
reading, math, writing, science, and social studies in grades four, eight 
and twelve. These test scores will provide the base for determining school 
success when the State Board for Elementary and Secondary Education begins, 
in the 1993-94 school year, to assess schools to determine whether students 
are meeting goals • • • Also, local boards of education will be required to 
publish annually in the newspaper a report on the performance of students 
in each district. (Miller, et al., 1991, p. 29) 

I have previously discussed the overall configuration of the Kentucky 
school accountability system (Petrosko, 1991). I will give only a brief 
description or the system here, stressing mostly the features of the system 
that make it unusual. The school accountability system (recently named KIRIS, 
Kentucky Instructional Results Information System) uses measurements of both 
cognitive and non-cognitive variables. 

Performance assessment of students is an important feature of school 
accountability. "Performance assessment" is one of several phrases have been 
used in recent years along with "direct assessment, " "authentic assessment, 11 
and "alternative assessment." As Worthen (1993) noted, "the s veral types of 
assessment all exhibit two central features: first, all are viewed as 
alternatives to traditional multiple-choice, standardized achievement tests; 
second, all refer to direct examination of student performance on significant 
tasks that are relevant to life outside school" (p. 445). 
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Prior to 1990, Kentucky used standardized multiple-choice tests for 
statewide assessment. In the 1980*s, the Comprehensive Tests of Basic Skills 
(CTBS) was used for several years. After that, a Kentucky adaptation of the 
Ci'BS was used. The passage of KERA in 1990 signaled a radical break with the 
use of such instruments. 

In accordance with KERA, a number of non-traditional measurements have 
begun being collected for the school accountability system. Students in grades 
4, 8, 12 and other gi -»des were tested in a variety of ways in the 1991-1992 
school year. In testing called "transitional, 11 students received multiple- 
choice and open-ended tests, and responded to a writing prompt. Tests had some 
common items and some matrix-sampled items. The open-ended items allowed 
students to construct responses to questions in their own words. Such items 
were scored by trained graders, and were consistent with the philosophy of 
performance assessment. 

Students were also tested with "performance events. 11 These involved having 
testers come to schools and examine students, who worked in small groups, with 
open-ended questions, problems, or tasks (in mathematics, science, and social 
studies). Finally, writing portfolios were collected from students. 

On each type of cognitive measurement, student performance was judged to be 
in one of four categories: a) novice, b) apprentice, c) proficient, 
d) distinguished. These designations were applied by comparing the student's 
performance to a scoring rubric. Each rubric had a description of the kinds of 
responses that would mevit a given designation. For example, a student who 
correctly answered a mathematics problem by correctly listing all steps for a 
solution might be given a proficient , but another student who missed steps or 
who made minor errors in steps would be rated apprentice . Levels of 
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performance were based on the opinions of subject matter experts and teachers. 
There was not a norm sample established, as would be for a traditional 
standardized test* 

In addition to the cognitive measures described above, non-cognitive 
variables were measured at all schools* These were: a) attendance rate, 
b) retention rate, c) dropout rate, and (at high schools) d) transition rate* 
Retention means percentage of students held back a year (a low percentage is 
better than a high percentage)* Transition rate means successful transition to 
adult life — the percentage of twelfth graders who go on to a job, the military, 
or further education rather than "graduating" to -the unemployment rolls* 

Using results from both the cognitive and non-cognitive measures, and 
going through a series of arithmetic calculations, each school was assigned an 
accountability index* Figure 1 shows what the source data would look like in 
calculating the index from a hypothetical high school* All of the numbers shewn 
would be weighted in various ways and subjected to additional calculations — 
eventually leading to one number, an accountability index* For the data in 
Figure 1, the index is 39*5* 
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Noncognitive Data 



Attendance Rale 


93.06% 


Retention Rate 


6.07* 


Dropout Rate 


7.52* 


Transition Rate 


85.11* 



Percentage of Students at Each Performance Level 



Type 


Level | 


Reading 


Math 


Science 


Soc. Stud. 




Common 
Open- 
ended 


Novice 1 


42.02 


56.28 


33.18 


46.51 


Writing | 


Apprentice | 


51.63 


37.20 


63.98 


44.91 




Proficient j 


5.60 


6.06 


2.35 


8.56 




Distinguished I 


0.75 


0.46 


0.49 


0.02 




Matrix- 
sampled 
Open- 
ended 


Novice H 43.77 


49.89 


31.06 


42.62 




Apprentice 


49.69 


40.00 


65.73 


46.71 




Proficient 


5.30 


8.00 


2.15 


8.67 




Distinguished 


1.24 


2.11 


1.06 


2.00 




Perform- 
ance 
Events 


Novice 




50.49 


39.70 


49.81 




Apprentice 




39.51 


45.77 


44.49 




Proficient 




7.94 


10.05 


4.10 




Distinguished 




2.06 


4.48 


1.60 




Writing 
Portfolio 


Novice 










• 33.96 1 


Apprentice 










50.14 | 


Proficient 










11.99 j 


Distinguished 


h- 








3.91 | 



Figure 1* Data from a high school that would be used to calculate the school's 



accountability index. (Source: Advanced Systems in Measurement and 
Evaluation, Inc., 1993b) 
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What happens to the accountability index? It is used in the final step of 
the process: computing a threshold , that is, a target for future performance. 
The process assumes that the theoretical ideal accountability index for a 
school is 100. In the hypothetical data presented above the index of 39.5 is 
60.5 different than the ideal (100 - 39.5 = 60.5). To set the threshold for 
the school, 10% of the discrepancy number is calculated. So 10% of 60.5 is 
approximately 6.1 accountability points. This is added to the school's 
accountability index, 6.1 + 39.5, and the result, 45.6, is the threshold value, 
which is a target for performance when the school is measured and tested over 
the next two years. In other words, when cognitive and non-cognitive measures 
are calculated over the next two years, the school will need to have a 45.6 
accountability index to meet its threshold value. 

Schools that improve Schools that are one or more points above threshold 
will have financial rewards, and school staff members will decide how those 
funds will be spent. 

Schools that stay the same If a school does not meet its threshold, but 
does not decline below its baseline accountability index, the school will be 
required to develop a school improvement plan and will be eligible to apply for 
money to implement the plan. 

Schools that decline somewhat Schools that decline less than five points 
below the baseline accountability index will be required to develop an 
improvement plan (for which it is eligible to receive funds). In addition, the 
school will get one or more Distinguished Educators to help school personnel. 
These educators, who are specially trained practitioners, will help school 
personnel write and implement the plan. 
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Schools that decline markedly A school more thati five ptrcent below 
baseline accountability will be declared "in crisis." People in the school 
will have to write a plan and will be helped by a distinguished educator (see 
the section above). In addition, the distinguished educator will have the power 
to retain, dismiss, and transfer school personne 1. Also, students may be 
allowed to transfer out of the school into a "successful" school, even it the 
latter is in another school district. 

In summary, the school accountability component of Kentucky Education 
Reform Act has several distinctive features. Performance testing is emphasized 
rather than multiple-choice testing. Student performance is described with the 
terras novice, apprentice, proficient, and distinguished rather than with 
standard scores like percentile ranks. For accountability, each school is a 
unit that will be compared against itself — with the school's performance in 
1991-1992 used no set performance targets for 1993-1994. Financial rewards 
will be given to schools that exceed targets and various sanctions are imposed 
on schools that decline in performance. Kentucky is unique in the nation for 
having, simultaneously: a) performance assessment as the backbone of student 
assessment, b) a "high stakes" accountability system with very clear and 
tangible consequences for various outcomes, and c) the school as the unit of 
educational change. 
KERA Evaluation Governing Board 

At the end of November 1992, the Governor of Kentucky announced the 
appointment of the KERA Evaluation Governing Board. This group of 10 private 
citizens is charged with forming a non-profit corporation that will raise funds 
from private and public sources tc support "an in-depth evaluation of the 
impact of KERA ... [which] should include but not be limited to the effect of 
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the reforms on students, individual schools, school systems and educators" 

(Jonas, 1992, p. 1). 

In the proposed operational plan for the evaluation, three basic question* 
are given as guiding purposes for this accountability effort. 

1. What are the intended and unintended effects of KERA on students? 

2. How were KERA and its components implemented, what factors affected 
implementation, and did implementation affect the educational system and 
student outcomes? 

3. How can the various benefits of KERA be enhanced and the implementation 
of its components be improved? 

The Evaluation Governing Board has $400,000 funding for its first year of 
operation (donated by a philanthropic foundation). The Board must hire an 
executive director and then begin developing a research agenda to carry out its 
charge • 
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Accountability for School Reform at the State Level 



Kentucky is certainly not unique in its efforts at statewide school 
reform* Recent years have seen a number of states trying to improve education 
by instituting curricular and organizational innovations. Programs and 
policies have included site-based management, statewide curriculum frameworks, 
integrated children's services, teacher career ladder programs, and schemes to 
financially reward schools that improve on outcome measures (Odden, 1992). 
( Side note . Kentucky has tried, or is now trying, all of the items in this 
list.) 

Educational reform efforts are largely funded by state tax revenues. Like 
other government programs, they require monitoring, oversight, and assessment. 
This is especially true when reform programs have been enacted by virtue of 
increased taxes. Quite naturally, members of the public want to know if their 
money is being spent wisely. Given this interest, what sorts of accountability 
mechanisms have been developed to assess school reform measures? 

In an interview study involving states in all regions of the U.S., 
Wohlstetter (1991) set out to describe exemplary accountability procedures, 
i.e., to describe "best current practices." Wohlstetter 1 s article provided a 
classification, a critique, and the basis for a theory regarding state 
accountability mechanisms for school reform. She identified four kinds of 
mechanisms that have emerged: a) Legislative oversight committee model, 
b) Executive branch model, c) Partnership model, and d) Third-party model. 
Brief descriptions of these are given below. 
Legislative oversight model 

Examples of the legislative oversight model are the Education-finance 
committee in Tennessee, the Legislative leader committee in Texas, and the 
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Legislative-audit committee of Virginia* Such committees consist of state 
legislators and have oversight as their sole purpose. They generally benefit 
from this concentrated focus, because they avoid the distractions of writing 
new legislation. 
Executive branch model 

With this model , overs ight is t ied more closely to the governor of the 
state. For example, the Connecticut State Board of Education has statutory 
responsibility for setting policy and monitoring the condition of education. 
Members of the nine-member board are laypersons appointed by the governor, but 
attempt to remain independent of both the governor and legislature* 
Partnership model 

In this context, partnership means a cooperative relationship between two 
or more separate organizational entities. One example is a school-business 
partnership in South Carolina. The "working arm" of the partnership is a 20 
member Business-Education committee consisting of 10 business and civic 
leadsrs, 6 educators, and 4 legislators. The group participates in evaluating 
a state education reform law. 

In the state of New York, partnership of different sort involves state 
government officials with those at the local level. The state Board of Regents, 
commissioner, and state department of education set broad policy guidelines and 
minimum achievement standards for local districts. The latter can operate 
their own accountability systems, as long as individual schools meet minimum 
state standards. 
Third-party model 

One example of this model of accountability is Policy Analysis for 
California Education (PACE). This is a university consortium that does policy- 
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relevant educational research. PACE involves professors from the University of 
California, Stanford University, The University of Southern California, and 
other institutions. It is funded by a private foundation and maintains 
independence from the governor, legislature, and state department of education. 

Another example of a third-party would be a outside organization (e.g., a 
private firm) hired to do performance auditing. The latter is analogous to 
financial auditing and would involve determining the level of implementation of 
reforms and measuring the extent to which program results have been achieved, 
Kentucky Accountability Mechanisms in Terns of the Vohlatetter Taxonoay 

Several accountability components of KERA fit the Wohlstetter classification 
of accountability mechanisms. First of all, Kentucky's Office of Educational 
Accountability (OEA) is considered an arm of the state legislature, and thus 
falls in the category legislative oversight model. As was stated earlier, 0E0 
is an independent arm of the legislature and is charged with monitoring the 
implementation of KERA, especially its governance provisions. 

State department of education bureaus fit under the executive branch model 
because they ultimately report to the Commissioner of Education and the State 
Board of Education, who are all gubernatorial appointees. A particular set of 
priorities of a governor can be reflected in the governor 1 s appointments and 
interaction with those in these offices. 

School accountability, with its system of rewards and sanctions, also best 
fits under the category executive branch model. Day-to-day operation of the 
assessment is direct ed by the private firm of Advanced Systems in Measurement 
and Evaluation under a contract from the department of education. The latter 
carries out policies of the State Board of Education and the Commissioner of 
Education. 
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The most recent accountability initiative is the KERA Evaluation Governing 
Board, announced at the end of November 1992. This group of 10 citizens is 
charged with "the development and implementation of an evaluation of the 
Kentucky Educational Reform Act 11 (Jones, 1992, p. 2). This will be 
accomplished by establishing a ion-profit organization, to be funded by 
foundations and sources other than state tax revenues. Clearly, the work 
Evaluation Board would be classified as a third-party model in the Wohlstetter 
taxonomy. 

After Wohlstetter (1991) described the four accountability mechanisms 
identified in her survey of states, she then discussed the uses of the data 
generated by the models and the consequences. In explaining the word use in 
this context, Wohlstetter drew upon the work of Carol Weiss. In a study of the 
operation of U. S. congressional committees, Weiss (1989) identified four kinds 
of functions that the information collected by committees can serve.. 

Support — certifies that the preexisting position is right; 

Warning — signals that a problem is (or i.s not) severe; 

Guidance — indicates better alternatives ; 

Enl ightenment--of f ers new constructs, new ways of thinking about issues. 

(Weiss cited in Wohlstetter, 1991, pp. 42-43) 
These categories of use can be applied to accountability mechanisms for 
state education reform. Indeed, Wohlstetter did this with the four model 
mechanisms she identified. I do something similar below. I offer a few 
comments below about what appears to be the major functions played by the 
accountability mechanisms that I have identified in Kentucky. Judgments are 
summarized in Table 4. 

Regarding Kentucky's Office of Educational Accountability (OEA), its most 
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obvious function seems to be warning of problems. Although this office is 
charged with several duties r the one that has been especially prominent has 
been investigating unresolved allegations of wrongdoing. Included in this list 
are several items related to governance, especially related to nepotism and 
financial favoritism (see Table 2). Guidance for policy decisions would also 
be part of its mission, but the overall charge of "monitoring the 
implementation of the Education Reform Act" is so large a task that, 
inevitably, other agencies have to be involved. 

There are two accountability mechanisms under the executive branch: 
department of education bureaus and the school accountability system. These 
both seem to have a primary purpose of providing guidance for policy decisions, 
and secondarily to be functioning to warn of problems and to develop support 
for current policies. 

Guidance for policy decisions might occur in several ways. On might be to 
present policy makers implementation data on the reform law and to influence 
policy changes on the basis of results obtained. For example, when the head of 
a bureau reports that schools implementing the ungraded primary program need 
more inservice for teachers, this would provide input on policies governing the 
kinds of professional development activities that should be planned. 

Warning of problems can occur when implementation data reveal that plans 
are not being carried out. Additionally, shortfalls in system performance are 
also warnings. When the KERA-mandated performance assessment occurs in each 
school, some schools will fall below their target threshold of performance, or 
even have a lower percentage of successful students than during baseline 
assessment. Being declared a "school in crisis" is about as dramatic a warning 
as can be. 
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It might seem peculiar to list "suppo r t for current policies" as a 
function of accountability mechanisms, but this is entirely realistic. KERA, 
like any school reform, depends for its survival on legislators and citizens 
perceiving its benefits . Many aspects of the law can be successfully 
implemented and it is entirely understandable that a public agency would want 
to collect data about success and to publicize this. 

The last accountability mechanism to be discussed is the KERA Evaluation 
Governing Board (and any organization that evolves from it). Here again, 
policy guidance seems to be a dominant function. This guidance could be very 
potent, because evaluation studies sponsored by a neutral, statewide board of 
citizens can have greater credibility than those sponsored by governmental 
agencies • 

How the KERA board will operate is speculation at this point because it is 
just being organized. However, a secondary function of "support for current 
policies" may emerge as important. Wohlstetter (1991) described how PACE, the 
California third-party organization, sponsored a study documenting a reform 
law's benefits for secondary schools: "PACE ' s findings . . . invigorated public 
interest in education in California and also provided ammunition to reform 
advocates' policy debates at the state level" (p. 44). 
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Table 4 

Organizations Involved with Kentucky School Reform Accountability 
Classified Using Models Identified by Wohlstetter (1991) 



Model and Kentucky 
Example 



Dominant 
Function 



Secondary 
Functions 



Legislative oversight 
model 

Office of Educational 
Accountability (0E0) 



Warning of 
problems 



Guidance for 
policy decisions 



Executive branch model 



Department of 
education bureaus 



Guidance for 
policy decisions 



Warning of 
problems 

Support for 
current policies 



School 

accountability 
system 



Guidance for 
policy decisions 



Warning of 
problems 

Support for 
current policies 



Third party model 

KERA Evaluation 
Governing Board 



Guidance for 
policy decisions 



Support for 
current policies 

Earning of 
problems 
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After examining various accountability mechanisms, Wohlstetter (1991) 
developed a set of criteria to judg.e their effectiveness. It is worth using 
these criteria to examine the strengths and weaknesses of the Kentucky 
mechanisms. In Wohlstetter 1 s view, effective accountability mechanisms should: 

1. Be empowered by state government, 

2. Have monitoring or oversight as a primary mission, 

3. Be independent from impleraentors , 

4. Have strong relationships with other policy actors and with leaders 
outside government, 

5. Communicate findings to multiple constituencies. (1991, pp. 45-46) 
Based on Wohlstetter s analysis of how the mechanisms she studied operate, and 
my knowledge of the Kentucky system, I offer some judgments (summarized in 
Table 5) of how KERA mechanisms meet these criteria. 

The Office of Educational Accountability (OEA) and department of education 
bureaus are alike in that both are clearly empowered by state government to 
engage in accountability activities, and both are at least "moderate" in the 
degree to which oversight is a primary mission. Regarding independence from 
implementors, OEA has the advantage here because the office is really a branch 
of a legislative service agency. The department of education does monitoring, 
but it also creates the very programs and regulations that it is monitoring. 
Both mechanisms would be moderate in terms of relationships to top policy- 
makers (being more directed to other practitioners than anyone else) and 
restricted to communicating with teachers, principals, and other educators than 
other audiences. 

I view the school accountability system to be moderate to low on most 
criteria. There is no doubt that the system is highly important to KERA, and 
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has been fully empowered by state government. However, the system, as it 
currently stands, has the most relevance for teachers and principals in the 
schools throughout the state* Its many technical components are hard to 
communicate to policy-makers or other audiences. 

Finally, the KERA Evaluation Board has much promise as an accountability 
mechanism. Unlike PACE in California, the KERA Board is fully empowered by 
state government (a gubernatorial order) and has oversight and evaluation as 
its sole mission. It is independent from implementors of KERA, and should have 
the ear of major policy makers. It is likely to view its mission as 
communicating with the many public school constituencies throughout the state. 
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Table 5 

Organizations Involved with Kentucky School Reform Accountability Rated on 



Criteria Identified by Wohlatcttcr (1991) 

Criteria 



Model and Kentucky 
Example 



Be 

empowered 
by state 

government 



Have over- 
sight as a 
primary 
mission 



Be 

independent 
from 

implementors 



Have 
strong 
relation- 
ships a 



Communicate 
to 

multiple 
groups 



Legislative oversight 
mode 1 

Office of Educational 
Accountability (OEO) 



High 



High 



Moderate 



Moderate 



Low 



Executive branch model 
Department of 

education bureaus High Moderate Low Moderate Low 



School 

accountability Moderate Moderate Moderate Low 

system Moderate 



Third party model 
KERA Evaluation 

Governing Board b U igh High High High High 



a Relationships with other policy actors and with leaders outside 
government • 

b Some judgments in this row are speculative, and based on experience of 
other states (e.g. California). 
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Future Directions for Accountability of KERA 

Kentucky's school reform law is about three years old. As has been 
discussed, there are several accountability mechanisms for the reform law that 
are operating, or planning to operate. The mechanisms have more than one 
function, and also have strengths and weaknesses. Do all of these things 
exhaust the possibilities of accountability? Not at all. In this paper, I 
purposely selected large-scale organizational efforts. A number of additional 
studies on KERA are *orth mentioning. 

Independent investigators, with no reporting responsibility to any state 
agency, have studied or are now studying, effects of the reform law. To cite 
one example, Coe, Kannapel, and Lutz (1991) reported on the initial phase of a 
qualitative study of the implementation of KERA in several rural school 
districts. The study is planned to continue until 1995. In addition, 
investigators at the state universities have been involved in several empirical 
studies. For example, under contract to the state department of education, 
researchers from the University of Kentucky analyzed KERA preschool and its 
collaborative relationship with already-existing Head Start programs. Also, 
researchers from the University of Louisville have surveyed elementary teachers 
on their reactions to ungraded primary school and surveyed special education 
teachers about the effects of school reform on special students. 

Is Kentucky doing enough in accountability? Could more be done? If these 
questions relate simply to the kinds of things that could be done, there is 
evidence that provisions of KERA are being evaluated with many of the 
techniques that experts have identified as useful. 

In a monograph sponsored by the U. S. Department o f Education, Kirst 
(1990) summarized "promising developments in accountability" throughout the 
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country. He listed six developments, and it is interesting to note that five 
of these are being implemented in Kentucky. Below are Kirst's accountability 
developments and one example that corresponds to each of them from the 
assessment of KERA. 



Kirst's accountability development 



1. Performance reporting 

Statewide measures of student 
outcomes 



Kentucky example 



School accountability system 
and threshold values for each 
each school 



2. Monitoring and compliance with 
standards or regulations 

Determining whether deserving 
children get services 



Office of Education 
Accountability (OEA) 



3. Incentive systems 

Rewards for educators for 
specific results 



School accountability system 
and potential financial 
rewards for school personnel 



4. Reliance on the market 

Voucher plans, magnet 
schools 



NO STATEWIDE IMPLEMENTATION 
OF THIS 



5. Changing the locus of authority 
or control of the school 

Parent advisory councils, 
community-controlled schools 



School-based decision-making 
involves a council with 
parents, teachers, and the 
principal 



6. Changing professional roles 

Boards for professional 
teaching standards 



Kentucky Education Professional 
Standards Board will establish 
requirements for obtaining and 
maintaining a teaching 
certificate 
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The only accountability development listed by Kirst not being used in 
Kentucky is "Reliance on the market. " However, although this is not being used 
statewide, it is having an impact in some districts within the state. For 
example, magnet schools are being extensively used in Jefferson County Public 
Schools, the largest district in the state (93,000 students), and the district 
that includes Louisville and surrounding suburbs. 

Taken as a whole, the array of assessment activities in the state is 
impressive. It might be tempting for Kentucky educators to view assessment 
related to KERA as totally sufficient. But no system of assessment and 
accountability is perfect. Kentucky would benefit by widening the scope of 
current assessment efforts and by improving the data linkages that exist among 
the efforts that are now going on. In summary, several - aspects of the 
assessment of KERA could be improved. Below are examples. 

1. An attempt should be made to provide performance data that are policy 
relevant. Kirst (1990) stated that accountability systems should highlight 
things that policy makers can change— teacher preparation, textbooks, number of 
science courses required. Assessment of KERA does not appear to be structured 
to do this. The school accountability system of KERA is elaborate and seems to 
have the potential for improving instruction at the individual school level. 
However, school accountability results and other variables, input variables 
like teacher preparation and process variables like course requirements, are 
not integrated into a single research database. It would be beneficial to tie 
input variables, process variables, and outputs together. In two years, some 
Kentucky educators will reap financial rewards because their schools exceed 
their "threshold" accountability index, and some schools will be declared "in 
crisis" because of poor performance. It would be useful to know what 
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systematic differences exist between such schools on potentially alterable 
variables . The kind of research needed in this area appears to be within the 
charge made to the KERA Evaluation Governing Board. The latter organization 
could provide an important service by sponsoring studies linking ? nput , 
process, and outcomes* 

2. Systematic research needs to be done on potential biases that could 
occur with the school accountability system, specifically the practice of using 
accountability index values and threshold values. As Kirst (1990) noted, M Not 
all schools start out at the same level in such areas as resources, pupil 
attainment, or teacher experience" (p. 16). According to KERA, each school 
will be compared against itself. This would seem to be eminently fair. 
However, the achievement of each school two years hence will be compared 
against a threshold value set last year. In rural communities, with little 
student mobility, the student body will be relatively stable year-by-year. As 
a result, comparing an accountability index from one time period to another 
will involve a similar student population. However, in urban areas, student 
mobility tends to be high. An accountability index for a school might depend 
on a large proportion of new students who transferred into the school since its 
threshold value was set. Rewarding or sanctioning teachers at a particular 
school does not make much sense if it is based on measuring the achievement of 
students who got most of their education at another school. 

3. In the long run, school reform in the state will have the best chance to 
succeed if Kentucky students are seen as comparing favorably with students in 
the rest of the country. At present, the question of how to compare students, 
or even whether to compare students, is unresolved. There are many innovative 
features in the student assessment being used for KERA. However, 
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the system is unique to the state. Some parents are already requesting 
comparative data regarding their child and children in other parts of the U.S. 
Moreover, the Kentucky Supreme Court decision that led to the passage of KERA 
specified that children in the state should receive an education that allows 
them to compare favorably with students in other states. The accountability 
system would be strengthened by allowing these kinds of comparisons. 

4. A number of issues related to the system of incentives and sanctions are 
worthy of investigation. For example, the weight of the school accountability 
system falls on teachers and school principals. These school personnel get 
financial rewards for improvements in school performance or get sanctions for 
poor school performance. Perhaps this is appropriate. However, this emphasis 
raises questions . about the role and responsibility of students and parents. It 
also raises issues about the functioning of the performance assessment system. 
How much test-taking motivation will there be if there are no immediate 
consequences, positive or negative, for students who are assessed with 
ccount ability measures? Kentucky is often cited for having a "high-stakes 11 
testing system, but most of the stakes are for school professionals, not 
students or parents. 

5. Some documentation needs to occur of what is actually going on in 
classrooms as a result of school reform. Performance assessment was emphasized 
in Kentucky because it was hoped that teachers would put more stress on 
problem-solving and other forms of student activity that require higher -level 
thinking. It is not unreasonable to suppose that teachers will indeed change 
their instructional practices to reflect the fact that portfolios and other 
kinds of complex student output will be used for school accountability. 
However, just what are teachers doing? And what will they be doing in the 
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future, as the reform proceeds? Related to this is the question are several 
others. Performance assessment is still a testing technology with questions 
about its reliability (Worthen, 1993)* There is little, if any, backlog of 
experience about what problems might arise when financial rewards or sanctions 
are tied to performance assessment. Research also needs to be done on how the 
curriculum, and recently adopted statewide curricular goals, impact upon 
classroom instruction and the performance of students on accountability 
assessments • 

6. A more experimentalist mind-set regarding school reform needs to be 
fostered within the state. This already exits in the area of performance 
assessment. It was frankly acknowledged from the beginning that statewide 
student assessment would evolve over time and might be changed as circumstances 
dictate. For example, it was originally believed that multiple-choice testing 
would be used for the indefinite future as a sort of backstop to performance 
assessment. However, data analysis of the 1991-1992 school year lad the 
contracted testing company to conclude that multiple-choice items could be 
quickly phased out and replaced by more "authentic" testing formats (Advanced 
Systems in Measurement and Evaluation, Inc., 1993a). 

It is inevitable that changes will be needed in other aspects of KERA. 
However, the open-minded spirit evident with performance assessment does not 
seem as evident with other parts of the reform law. Within the current media 
and education establishment in Kentucky, criticism of any part of KERA seems to 
be equated with being anti education-in-general. One factor that may explain 
this reluctance to re-think parts of the law is a fear that the whole thing 
will unravel if changes are allowed in any part of it. This thinking could be 
a carry-over from the days when there were legislative struggles to pass the 
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law in the first place. When KERA was debated by the state legislature in 
1990, the law was presented for consideration as a complete package. A 
straight up or down vote determined the fate of the law. This probably made 
good legislative sense. However, once the law has been allowed to function, it 
is not very realistic to view it as a sacrosanct unitary package that cannot be 
altered in any way. This is especially true given the fact that KERA is such a 
large collection of different (and somewhat independent) programs and policies. 

Another factor that may inhibit thoughtful debate about KERA is that the 
law contains just about every "educationally correct" idea that was in the air 
in the late 1980 V. Perhaps people do not want to believe that any of these 
good ideas could possibly fail. There are fallacies here of course. Even 
great ideas sometimes fail (or fall short of expectations) when someone tries 
to implement: them outside the context in which they were developed. 
Futhermore, a complex law like KERA is a unique combination of separate reform 
ideas, some of which have never been tried together at once. Some parts of the 
reform law may clash with or be incompatible with other parts of the law. 

Certain fundamental organizational and sociological tensions have not been 
eliminated by the reform law, they are simply operating within a new framework. 
The issue of locus of control of education is one example. There is always 
some tension in education between central authority and the autonomous 
decisions of the individual teacher in his or her classroom. This tension is 
still very much present in Kentucky. What has been added to the scenario are 
some new organizations, e.g., school councils consisting of teachers, parents, 
and the principal, that are now involved in educational decision-making. 
School councils have to co-exist with school boards, the state department of 
education, teacher organizations, and a variety of other groups and 
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organizations that impact upon the teacher and the world of the classroom. 

Despite some of the challenges outlined above, KERA remains a model of 
broad-scope state education reform. KERA and its related accountability 
activities can provide educators with a rich source of data on school change. 
By coordination of already planned assessment activities, the addition of new 
activities, and adopting a pragmatically experimentalist spirit, much can be 
learned about fundamental change of elementary and secondary education across 
an entire state. 
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